🕷️️ Job Radar • SCRAPING

freelancer.com 🟡 2026-05-22

🔹 Automated Daily Multi-Source JSON Data Extraction Pipeline
👤 Client: 🇮🇳 Amritsar, India Member since 2018-06-14
💰 Price: $12 Average bid
🚩 Problem: Need for a resilient, automated scraping system to handle rotating target sites and deliver structured data into existing pipelines.
📦 Existing: List of target URLs and required field specifications.

Specifications:

[Target] Rotating set of web-data sources
[Method] Automated extraction with CAPTCHA and IP block bypass
[Stack] Python, Scrapy, BeautifulSoup, Selenium
[Security] Secure credential management via Git
[Format] Structured JSON with matching field hierarchy
[Format] Success/Failure logs and summary reports

Workflow:

1. Implementation of spiders/scripts based on provided URL lists and field maps.
2. Integration of retry logic and error handling for layout changes and blocks.
3. Deployment of unattended scheduling via cron.
4. Delivery of parsed JSON and execution logs to data pipelines.